Skip to content

[fix](paimon) Harden JNI reader options and transport - #66247

Open
Gabriel39 wants to merge 6 commits into
apache:masterfrom
Gabriel39:agent/paimon-reader-guardrails
Open

[fix](paimon) Harden JNI reader options and transport#66247
Gabriel39 wants to merge 6 commits into
apache:masterfrom
Gabriel39:agent/paimon-reader-guardrails

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

The Paimon JNI read path had several correctness and operational guardrail gaps:

  1. read.batch-size accepted zero, negative, or excessively large values. In particular, zero can make the vectorized reader return without advancing input.
  2. scan.manifest.parallelism=0 can block on a zero-permit semaphore, while a value above the CPU-sized default causes Paimon to replace a JVM-static executor and leaves the new pool visible to later queries.
  3. JNI projected fields used delimiter-based framing. A legal quoted column such as `region,code` was decoded as two fields, and nested STRUCT child names containing #, ,, or : also collided with the type grammar.
  4. The legacy Paimon reader selected by enable_file_scanner_v2=false did not emit the safe schema payload used by FileScannerV2.
  5. DEBUG logging printed the complete scanner parameter map, which may contain storage or JDBC credentials.
  6. Catalog property validation temporarily published the candidate map. A concurrent first query could initialize from unjournaled values that were subsequently rejected and rolled back.
  7. Catalog-level paimon.table-option.* accepted options which can select a different table context or affect writes after Doris has already bound the original schema.
  8. Tightening that allowlist could make an existing catalog unloadable if its persisted image already contained a formerly accepted option.
  9. Physical Paimon table options bypassed Doris catalog/relation validation and could still carry unsafe batch size, async threshold, or manifest parallelism into planning.
  10. Every scanner statistics collection enumerated all JVM threads and allocated a ThreadInfo[], although the Paimon async-reader pool is JVM-global.
  11. Safe reader tuning could not be supplied for an individual table@options(...) relation.

How does this PR solve it?

  • Introduces one shared reader-option validator and allows only:
    • read.batch-size: 1..65536
    • file-reader-async-threshold: 1 MB..1 GB
  • Applies explicit precedence without mutating the physical Paimon table:
    relation @options > Doris catalog option > physical table option > Paimon default.
  • Validates the final physical/catalog table view before planning or serialization. This prevents a table created with read.batch-size=0 or scan.manifest.parallelism=0 from bypassing Doris-side safeguards.
  • Bounds relation-scoped and physical manifest parallelism to 1..Runtime.availableProcessors(), preserving the JVM-global executor-capacity invariant.
  • Emits paired required_fields_base64 and columns_types_base64 schema payloads from both the legacy and V2 C++ readers. Every complete list item is independently encoded, and every nested STRUCT child name is encoded inside the type descriptor.
  • Keeps the legacy required_fields and columns_types parameters as a compatibility fallback. The Java scanner accepts only a complete encoded pair, so it cannot combine two framing versions accidentally.
  • Replaces raw parameter logging with a fixed summary containing only batch size and projected-field count.
  • Adds detached catalog-property validation. Paimon validates a merged candidate CatalogProperty before the live catalog is changed; connectors without this capability retain the existing tentative-validation path and rollback behavior.
  • Rejects new unsafe CREATE/ALTER settings, but load-time reconstruction ignores rather than applies unsupported or invalid legacy paimon.table-option.* entries. The raw persisted property is retained so image/edit-log compatibility is not destructive.
  • Shares one async-reader thread-count sample across scanners for one second.

For example, the following nested schema is now transported without colliding with either the outer list framing or the STRUCT grammar:

CREATE TABLE quoted_fields (
  `nested#value` STRUCT<
    `hash#name`: STRING,
    `region,code`: STRING,
    `colon:name`: STRING
  >
);

Example query-level tuning:

SELECT id
FROM paimon_table@options(
    'read.batch-size' = '4096',
    'file-reader-async-threshold' = '32 MB');

Two aliases of the same table may use different values in the same statement because each relation receives its own Table.copy(...):

SELECT small.id
FROM paimon_table@options('read.batch-size' = '1') small
JOIN paimon_table@options('read.batch-size' = '8192') large
  ON small.id = large.id;

Thread statistics microbenchmark

Dataset and method:

  • One JVM process per measurement.
  • 16, 128, or 512 idle worker threads whose names match the Paimon async-reader prefix. Including JVM service threads, the observed live-thread counts were 22, 134, and 518.
  • 200 warm-up calls followed by 2,000 measured statistics calls.
  • Old path: every call executes getAllThreadIds() and getThreadInfo(ids, 0), then scans the returned ThreadInfo[].
  • New path: the exact one-second shared-cache algorithm used by the scanner; the cache is primed before measurement.
  • Elapsed wall time and bytes allocated by the calling thread were measured with ThreadMXBean. A checksum blackhole consumes every returned count.
Named workers Live JVM threads Old time / 2,000 calls Cached time / 2,000 calls Old allocated bytes Cached allocated bytes
16 22 41.507 ms 0.429 ms 12,600,576 0
128 134 139.544 ms 0.417 ms 71,430,512 0
512 518 581.066 ms 0.398 ms 274,048,216 0

This microbenchmark isolates statistics collection rather than scan I/O. It shows that the old cost grows with JVM thread count, while cache hits avoid both the full thread walk and its per-call allocation. The tradeoff is that this profile gauge can be up to one second stale; it does not affect query execution or scheduling.

Tests

Passed locally:

  • PaimonJniScannerTest: 15 tests, 0 failures, Maven build cache disabled.
  • FE checkstyle: 0 violations.
  • Doris C++ clang-format 16 check.
  • Groovy compilation of the P0 regression suite.
  • Paimon 1.3.1 compile/runtime probes covering bounds, physical manifest rejection, and legacy-option filtering.
  • Thread-statistics microbenchmark described above.

Added coverage:

  • BE unit tests for delimiter-safe JNI field-list framing and nested STRUCT type descriptors.
  • Scanner unit tests for top-level and nested quoted identifiers, paired schema decoding, real DEBUG log capture/redaction, and shared cache TTL.
  • FE unit tests for detached ALTER validation, a latch-controlled ALTER/initialization race, catalog/relation/physical option bounds, unsafe-option rejection, manifest parallelism, relation-local copies, and image/edit-log legacy-option reconstruction.
  • P0 external-Paimon regression for both V1 and V2 scanner routes; quoted comma/hash/colon/space/Unicode identifiers; nested STRUCT projection; physical unsafe options; relation precedence; independent aliases; invalid boundaries; unsafe catalog options; and failed-ALTER rollback.

Not run locally:

  • Full FE/BE compilation, FE/BE unit-test binaries, and the external-Paimon P0 environment. The checkout does not contain the required installed third-party toolchain (protoc included). CI buildall and regression checks are requested on this PR.

Release note

Paimon catalogs and relation @options now support bounded read.batch-size and file-reader-async-threshold reader tuning. Unsafe catalog, physical-table, and manifest-parallelism settings are rejected. Both Paimon scanner routes support quoted top-level and nested identifiers containing delimiter characters. Scanner DEBUG logs no longer include raw parameters, failed Paimon catalog validation is atomic, persisted legacy options remain loadable without being applied, and scanner statistics reuse a short-lived JVM-wide thread-count sample.

Check List (For Author)

  • Test
    • Unit Test
    • Regression test case added
  • Behavior changed:
    • Paimon dynamic reader settings now use an explicit safe allowlist and documented bounds.
  • Does this need documentation?
    • Yes. Supported options, precedence, bounds, examples, compatibility behavior, and the statistics-cache tradeoff are documented above.

Review follow-up

The review fixes make empty identifiers and option precedence explicit:

  • Every outer JNI schema token now carries a $ marker. Therefore an empty projection is encoded as the empty payload, while one empty field name is encoded as $; both the legacy and V2 C++ producers use this shared encoding.
  • Effective reader options are validated only after the complete copy chain is formed. For example, physical read.batch-size=0 plus catalog 1024, or plus relation @options(...=4096), is accepted with the safe final value; the same physical table without an override is still rejected. The same rule applies to manifest parallelism.
  • Added {} versus {\"\"} unit coverage, scanner constructor coverage, final-boundary/no-override coverage, safe catalog/relation override coverage, and P0 queries through both scanner routes. PaimonJniScannerTest now runs 15 tests with no failures. A direct Paimon 1.3.1 runtime probe also accepted the safe relation override and rejected the unsafe final value.

Review follow-up: hidden planning handles

This update closes the four remaining pre-ScanNode planning gaps:

  • Read-only system-table wrappers: $partitions exposes an empty option map while its private data table performs manifest planning. Doris now applies and validates relation options on that hidden data table, while type inspection remains non-planning so a safe override is not rejected prematurely.
  • Fallback branches: FallbackReadFileStoreTable is validated recursively because its main and fallback children plan independently.
  • Snapshot partition binding: partition enumeration now uses the already merged effective table handle. A physical scan.manifest.parallelism=0 is rejected before enumeration, while @options('scan.manifest.parallelism'='1') is preserved instead of being lost through a raw catalog reload.
  • Row counts and statistics: fetchRowCount() validates immediately before newScan().plan(); coverage exercises both ExternalRowCountCache and manual analysis entry points.

For example, given a partitioned table whose physical option is unsafe:

SELECT count(*)
FROM unsafe_table$partitions;
-- rejected before manifest enumeration

SELECT count(*)
FROM unsafe_table$partitions
@options('scan.manifest.parallelism'='1');
-- succeeds because the relation copy overrides the physical value

Added tests use real Paimon 1.3.1 PartitionsTable, FallbackReadFileStoreTable, and file-store table objects, plus a partitioned P0 fixture. FE checkstyle, Groovy compilation, and isolated Paimon fixture probes pass locally. Full FE unit tests and the external P0 environment remain delegated to CI because this checkout does not contain the installed third-party toolchain, including protoc.

Review follow-up: parser compatibility and wrapper identity

This update addresses four compatibility and planning-identity findings:

  • Legacy STRUCT compatibility: the public unencoded parser again normalizes child keys to lowercase. For example, struct<Mixed:int> produces the historical key mixed; only the versioned Base64 Paimon grammar preserves Mixed.
  • Privilege delegates: effective-table validation now walks every DelegatedFileStoreTable.wrapped() handle as well as the fallback child. Therefore a PrivilegedFileStoreTable(FallbackReadFileStoreTable(...)) cannot hide an unsafe fallback manifest executor.
  • System-table identity: system wrappers are constructed with Paimon's SystemTableLoader directly from the same cached FileStoreTable that Doris validates. The catalog is no longer called a second time, so a refreshed physical handle cannot be planned while an older handle is validated.
  • Reader-only projection reuse: relation options containing only read.batch-size and/or file-reader-async-threshold validate their effective table and then reuse the memoized latest snapshot projection. Snapshot-selecting and manifest-planning options retain the effective-handle projection path.

The partitioned counting test verifies that reader-only tuning performs zero direct projection loads, while the existing manifest-override test verifies that safety-sensitive options still load from the effective copy.

Validation for this update:

  • Targeted legacy parser test: 1 passed, with recorded RED output [Mixed, UPPER] before the fix and GREEN output [mixed, upper] after it.
  • PaimonJniScannerTest: 15 passed, confirming encoded nested names still preserve their exact spelling.
  • Real Paimon 1.3.1 privilege-wrapped fallback probe: rejected the hidden unsafe fallback after the fix.
  • FE and Java-extension checkstyle: 0 violations.

Full FE unit execution remains delegated to CI because the checkout does not contain the installed third-party toolchain, including protoc.

Review follow-up: late serialization and connector-scoped schema framing

This update closes the four findings from the latest review:

  • System-table descriptor serialization is non-planning. Schema construction reads the cached raw wrapper only for its row type, so a safe relation override is not replaced by validation of the unsafe physical handle after binding.
  • Memoized partition projection validates only manifest-planning settings. Reader batch and async settings are still validated on the final effective scan handle, but raw metadata loading no longer rejects values already replaced by a safe relation copy.
  • V2 encoded schema framing is now an explicit Paimon reader capability. JDBC, Hudi, Trino, Iceberg system-table, and MaxCompute readers skip recursive encoded-type construction, Base64 encoding, and the two Paimon-only parameter-map entries.
  • Every successful guardrail query now has a named result assertion. Coverage records quoted and nested values for both scanner routes, the empty identifier, catalog and relation overrides, alias isolation, failed-ALTER recovery, partitioned reader-option composition, manifest override, and system-table descriptor serialization.

For example, this composition now reuses the cached partition projection and validates the safe final scan handle:

SELECT *
FROM partitioned_table@options('read.batch-size'='4096');
-- physical read.batch-size=0 is ignored by metadata enumeration;
-- the effective relation copy is validated before data scanning

Validation for this update:

  • PaimonJniScannerTest: 15 passed.
  • FE checkstyle: 0 violations.
  • Regression framework compilation and Doris clang-format 16 checks passed.
  • FE/BE unit coverage and external-Paimon P0 result coverage were added but could not be executed locally because the installed third-party toolchain, including protoc, is absent; CI buildall and review were requested after the push.

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Paimon dynamic reader settings accepted unsafe or unbounded values, query-scoped
settings could resize a JVM-global executor, and failed catalog validation could
leave tentative properties installed. JNI projection framing also split quoted
column names containing commas, while debug logging and scanner statistics could
expose raw credentials or repeatedly scan every JVM thread.

This change introduces a reader-only option allowlist and bounded validation,
keeps catalog and relation precedence explicit, rolls back every catalog
validation failure, and transports projected identifiers with per-field Base64
framing. It also replaces raw parameter logging with a fixed safe summary and
shares JVM thread-count samples for one second.

### Release note

Paimon catalogs and relation @options support bounded read.batch-size and
file-reader-async-threshold tuning. Unsafe catalog table options are rejected,
manifest planning parallelism is bounded by JVM processor capacity, and JNI
projection now supports quoted identifiers containing delimiters.

### Check List (For Author)

- Test
    - Unit Test
    - Regression test
- Behavior changed:
    - Paimon dynamic reader settings now use an explicit safe allowlist and bounds.
- Does this need documentation?
    - Yes. This PR documents the supported query options in its description.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: five blocking correctness and compatibility issues remain in the Paimon hardening paths.

Checkpoint conclusions:

  • Goal and proof: the patch improves option validation, JNI identifier framing, raw-parameter log redaction, ALTER rollback, and thread-count sampling, but it does not accomplish the goal on all supported inputs and paths. The added tests do not cover the five blockers.
  • Scope and clarity: the change is generally focused and reuses shared validators, although the generic CatalogMgr rollback change broadens the impact of connector-specific validation.
  • Concurrency: the cached thread counter has sound volatile publication, synchronized refresh, retry, and wrap-safe TTL behavior. Failed ALTER is not atomic against concurrent lazy catalog initialization (MAIN-4).
  • Lifecycle: Paimon Table.copy is relation-local and does not mutate cached base-table options; no C++ static-initialization-order or ownership issue was found. Failed ALTER can nevertheless leave initialized derived catalog state alive after raw-map rollback (MAIN-4).
  • Dynamic options/configuration: catalog and relation reader overrides are observed without restart and their new bounds are clear, but equivalent values inherited from the physical Paimon table bypass validation (MAIN-3).
  • Compatibility: retaining the legacy JNI field is rolling-upgrade-friendly only on the V2 producer. The still-supported V1 producer omits the safe payload (MAIN-1), nested type framing remains delimiter-unsafe (MAIN-2), and persisted pre-upgrade catalog options can prevent initialization (MAIN-5).
  • Parallel and conditional paths: V2, V1/load routing, catalog, relation, and physical option sources were traced. The missed V1 and physical-table paths are blockers; the added validation conditions and error conversions are otherwise understandable.
  • Tests/results: helper unit tests cover basic parsing, validation, redaction, and cache TTL. The P0 success blocks assert only absence of exceptions, leave Scanner V2 enabled, and omit nested names, unsafe physical options, a latch-driven ALTER race, and image/edit-log upgrade coverage. No generated expected-result file was added.
  • Observability: raw scanner parameters are no longer logged and the cached JVM thread gauge is safe; no additional metric or log blocker was found.
  • Persistence/transactions/data writes: edit-log ordering is correct in the single-threaded success/failure paths and no data-write transaction path changes here. Concurrent failed ALTER can expose unjournaled derived state (MAIN-4), while reload of existing persisted properties is incompatible (MAIN-5).
  • Cross-module transport: no new FE-to-BE variable is introduced, but the BE-to-Java JNI schema parameter was updated in only one producer and its parallel type payload remains unsafe (MAIN-1 and MAIN-2).
  • Performance: the JVM thread-count cache removes repeated thread walks without a material lock bottleneck. Unsafe physical manifest parallelism can still create a zero-permit wrapper or enlarge Paimon's JVM-static executor (MAIN-3).
  • Additional user focus: no extra focus was supplied; the entire changed-file set was reviewed.

Validation was static only: the authoritative review prompt prohibits builds/tests, and this checkout lacks .worktree_initialized and thirdparty/installed. No build or test result is claimed.

Comment thread be/src/format_v2/jni/jni_table_reader.cpp Outdated
Comment thread be/src/format_v2/jni/jni_table_reader.cpp
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 89.66% (52/58) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 90.91% (10/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.04% (25557/43286)
Line Coverage 43.12% (256238/594184)
Region Coverage 38.81% (202996/523029)
Branch Coverage 40.14% (92549/230539)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29850 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17811	4213	4136	4136
q2	2025	337	201	201
q3	10249	1400	816	816
q4	4683	476	361	361
q5	7545	858	574	574
q6	190	181	142	142
q7	755	811	618	618
q8	9331	1659	1683	1659
q9	5626	4430	4345	4345
q10	6717	1745	1482	1482
q11	512	360	321	321
q12	742	577	470	470
q13	18098	3430	2760	2760
q14	267	274	255	255
q15	q16	787	790	716	716
q17	989	1086	988	988
q18	7243	5751	5533	5533
q19	1289	1269	986	986
q20	802	689	598	598
q21	6196	2591	2583	2583
q22	420	357	306	306
Total cold run time: 102277 ms
Total hot run time: 29850 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4457	4354	4393	4354
q2	280	312	219	219
q3	4550	4948	4387	4387
q4	2063	2118	1380	1380
q5	4420	4280	4299	4280
q6	235	182	131	131
q7	1727	2331	1669	1669
q8	2574	2222	2184	2184
q9	8004	7973	7755	7755
q10	4646	4629	4213	4213
q11	619	460	391	391
q12	732	759	533	533
q13	3295	3643	2899	2899
q14	313	310	294	294
q15	q16	707	731	669	669
q17	1356	1359	1320	1320
q18	7976	7404	7181	7181
q19	1171	1110	1089	1089
q20	2195	2201	1933	1933
q21	5194	4523	4423	4423
q22	511	469	395	395
Total cold run time: 57025 ms
Total hot run time: 51699 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177722 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

query5	4330	644	481	481
query6	448	220	205	205
query7	4843	602	376	376
query8	332	198	171	171
query9	8783	4118	4105	4105
query10	466	360	304	304
query11	5873	2321	2120	2120
query12	159	101	98	98
query13	1280	629	458	458
query14	6255	5281	4981	4981
query14_1	4319	4271	4247	4247
query15	213	200	177	177
query16	1006	484	428	428
query17	929	681	567	567
query18	2428	474	362	362
query19	207	187	144	144
query20	111	109	106	106
query21	233	159	139	139
query22	13670	13600	13441	13441
query23	17501	16559	16155	16155
query23_1	16259	16150	16177	16150
query24	7512	1784	1290	1290
query24_1	1310	1305	1273	1273
query25	540	444	382	382
query26	1330	342	216	216
query27	2633	570	383	383
query28	4535	2032	2020	2020
query29	1083	672	464	464
query30	331	267	228	228
query31	1128	1091	993	993
query32	111	59	59	59
query33	535	304	259	259
query34	1174	1170	638	638
query35	763	788	687	687
query36	1041	1045	868	868
query37	146	106	92	92
query38	1861	1709	1653	1653
query39	876	871	844	844
query39_1	841	834	825	825
query40	253	164	142	142
query41	71	64	64	64
query42	92	91	89	89
query43	336	335	287	287
query44	1479	779	752	752
query45	194	180	171	171
query46	1053	1180	751	751
query47	2128	2079	1987	1987
query48	409	421	303	303
query49	567	418	315	315
query50	1043	427	353	353
query51	10723	10756	10631	10631
query52	85	86	75	75
query53	276	281	210	210
query54	287	261	240	240
query55	81	71	67	67
query56	326	286	283	283
query57	1303	1280	1208	1208
query58	282	256	252	252
query59	1596	1651	1448	1448
query60	303	276	247	247
query61	183	147	150	147
query62	538	497	433	433
query63	243	196	207	196
query64	2829	1029	847	847
query65	4741	4630	4655	4630
query66	1853	507	374	374
query67	29262	29211	29124	29124
query68	3046	1489	1032	1032
query69	399	316	268	268
query70	923	816	856	816
query71	374	351	323	323
query72	3080	2834	2519	2519
query73	842	765	462	462
query74	5080	4954	4710	4710
query75	2546	2526	2141	2141
query76	2317	1193	823	823
query77	366	387	293	293
query78	11867	11873	11284	11284
query79	1259	1200	786	786
query80	652	574	502	502
query81	494	338	303	303
query82	242	159	124	124
query83	330	340	301	301
query84	319	164	134	134
query85	973	606	527	527
query86	298	253	233	233
query87	1817	1822	1759	1759
query88	3740	2830	2784	2784
query89	414	383	333	333
query90	2137	208	201	201
query91	201	188	161	161
query92	65	62	53	53
query93	1517	1546	960	960
query94	536	366	301	301
query95	829	507	495	495
query96	1009	810	358	358
query97	2620	2602	2471	2471
query98	206	208	200	200
query99	1089	1106	987	987
Total cold run time: 261851 ms
Total hot run time: 177722 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.04
query3	0.28	0.14	0.14
query4	1.60	0.14	0.14
query5	0.24	0.22	0.22
query6	1.21	1.06	1.05
query7	0.04	0.00	0.01
query8	0.05	0.04	0.03
query9	0.38	0.30	0.31
query10	0.53	0.54	0.54
query11	0.19	0.13	0.14
query12	0.18	0.14	0.14
query13	0.48	0.47	0.48
query14	1.01	0.99	1.01
query15	0.62	0.60	0.63
query16	0.31	0.33	0.34
query17	1.12	1.13	1.09
query18	0.22	0.20	0.20
query19	2.09	1.96	1.91
query20	0.01	0.02	0.01
query21	15.45	0.20	0.13
query22	4.94	0.05	0.06
query23	16.13	0.30	0.11
query24	2.96	0.41	0.32
query25	0.10	0.05	0.05
query26	0.72	0.20	0.14
query27	0.03	0.04	0.04
query28	3.53	0.88	0.56
query29	12.47	4.12	3.33
query30	0.27	0.15	0.15
query31	2.77	0.57	0.31
query32	3.22	0.59	0.48
query33	3.18	3.14	3.23
query34	15.62	4.24	3.53
query35	3.51	3.55	3.56
query36	0.55	0.43	0.41
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.04	0.04
query40	0.19	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.65 s
Total hot run time: 25.02 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (11/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.64% (31937/42222)
Line Coverage 60.27% (355707/590220)
Region Coverage 56.91% (298647/524790)
Branch Coverage 58.27% (134367/230593)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/58) 🎉
Increment coverage report
Complete coverage report

Use paired delimiter-safe schema framing in both scanner routes, validate effective physical options, validate Paimon ALTER candidates off the live catalog, and preserve load compatibility for legacy catalog options. Add unit and P0 coverage for each review scenario.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29454 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18006	4149	4139	4139
q2	2322	327	197	197
q3	10296	1478	829	829
q4	4687	466	347	347
q5	7672	844	572	572
q6	194	171	140	140
q7	773	837	622	622
q8	9978	1442	1519	1442
q9	6693	4343	4308	4308
q10	7803	1712	1463	1463
q11	505	343	317	317
q12	745	608	473	473
q13	18114	3309	2713	2713
q14	267	258	255	255
q15	q16	789	781	709	709
q17	972	1047	1030	1030
q18	6787	5703	5473	5473
q19	1337	1244	1136	1136
q20	789	684	591	591
q21	5662	2570	2407	2407
q22	413	353	291	291
Total cold run time: 104804 ms
Total hot run time: 29454 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4413	4332	4314	4314
q2	283	319	206	206
q3	4530	4948	4376	4376
q4	2014	2153	1355	1355
q5	4426	4231	4448	4231
q6	231	266	218	218
q7	1788	1768	1467	1467
q8	2332	2006	2042	2006
q9	7687	7566	7685	7566
q10	4703	4615	4235	4235
q11	530	392	413	392
q12	766	752	541	541
q13	3222	3527	2991	2991
q14	294	319	280	280
q15	q16	721	733	637	637
q17	1352	1307	1401	1307
q18	7360	6899	6934	6899
q19	1119	1091	1112	1091
q20	2255	2247	1961	1961
q21	5549	4832	4895	4832
q22	539	475	444	444
Total cold run time: 56114 ms
Total hot run time: 51349 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blocking correctness gaps remain:

  1. The new Base64 list protocol still maps one empty top-level identifier to the same payload as zero projected fields, so the Java constructor rejects a reachable external Paimon schema on field/type cardinality.
  2. Physical-plus-catalog options are validated before relation-scoped options are copied, so a safe relation override cannot take the documented precedence over an unsafe physical value.

Critical checkpoint conclusions:

  • Goal and proof: the PR substantially adds bounded Paimon reader options, physical/catalog validation, relation-local copies, delimiter-safe transport, log redaction, detached ALTER validation, replay compatibility, and cheaper metrics. The two findings above prevent the full advertised identifier and precedence behavior. Existing tests do not cover either failing case.
  • Scope and clarity: production changes are focused on Paimon/JNI behavior plus one generic ExternalCatalog validation hook and one shared JNI type parser extension. No unrelated source changes were found.
  • Concurrency and locking: detached Paimon validation does not publish tentative CatalogProperty state; accepted updates reset synchronized ExternalCatalog objects and invalidate Paimon caches. The global thread counter uses synchronized refresh and volatile publication with intentional one-second staleness. No additional deadlock or stale-publication defect was found.
  • Lifecycle and error handling: V1/V2 scanner construction and existing JNI cleanup remain aligned; mixed encoded schema halves fail loudly. Catalog rollback/replay/reset paths preserve prior or journaled state. No new resource leak or swallowed status was substantiated.
  • Configuration and dynamic behavior: no Doris runtime config item is added. Catalog ALTER resets derived state, relation options use per-handle Table.copy, and unsafe values are range-checked; MAIN-2 is the remaining ordering error in the precedence chain.
  • Compatibility and parallel paths: legacy schema parameters remain as fallback, V1 and V2 now emit the paired protocol, and persisted legacy catalog keys remain raw but inactive. Ordinary, branch, snapshot, system-table, cache-loader, planning, and JNI serialization paths were traced. MAIN-1 affects both scanner routes; MAIN-2 affects ordinary and system-table relation copies.
  • Persistence, transactions, and writes: the catalog edit-log replay path was checked and retains loadability without applying unsafe legacy options. The PR does not alter Doris data transaction or storage-format semantics.
  • Testing: review-time builds/tests were forbidden by the authoritative prompt and none were run in this review environment. CI was observed separately immediately before submission; its evolving state is not claimed as review-time test evidence. Changed tests cover the existing five review fixes but omit {} versus {""} framing and safe relation override of an unsafe physical value.
  • Observability and performance: raw credential-bearing scanner parameter logging is removed, fixed summary fields remain, and the JVM thread-count sample removes repeated ThreadMXBean allocation. No additional observability or material performance regression was found.
  • FE/BE propagation and nullability: both legacy and V2 BE producers publish the new paired Java parameters; no new FE-to-BE thrift variable or nullable-column manipulation is introduced.
  • User focus: no additional user-provided focus was supplied; the full PR was reviewed.

The five existing inline threads were rechecked and not duplicated: legacy V1 producer parity, nested type framing, no-override physical option validation, detached ALTER validation, and persisted-catalog compatibility are addressed at the reviewed head.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177617 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

query5	4317	634	477	477
query6	515	240	203	203
query7	4935	591	351	351
query8	368	191	167	167
query9	8792	4027	4046	4027
query10	461	361	291	291
query11	6248	2361	2106	2106
query12	158	105	103	103
query13	1359	649	467	467
query14	6737	5280	4966	4966
query14_1	4342	4239	4229	4229
query15	217	212	189	189
query16	1040	475	441	441
query17	1130	706	581	581
query18	2684	468	350	350
query19	227	194	155	155
query20	118	110	109	109
query21	244	158	137	137
query22	13704	13610	13344	13344
query23	17375	16524	16716	16524
query23_1	16297	16406	16445	16406
query24	8069	1813	1343	1343
query24_1	1334	1366	1367	1366
query25	585	498	428	428
query26	1552	376	228	228
query27	2838	632	375	375
query28	4376	2012	1990	1990
query29	1073	579	467	467
query30	345	258	220	220
query31	1122	1084	969	969
query32	98	58	56	56
query33	530	312	250	250
query34	1223	1127	653	653
query35	831	769	655	655
query36	989	1024	885	885
query37	156	107	86	86
query38	1878	1726	1642	1642
query39	884	865	836	836
query39_1	839	864	846	846
query40	253	166	140	140
query41	65	61	60	60
query42	89	88	87	87
query43	338	340	278	278
query44	1418	759	742	742
query45	187	175	165	165
query46	1070	1250	751	751
query47	2080	2085	1930	1930
query48	380	406	281	281
query49	593	407	298	298
query50	1097	455	341	341
query51	10695	10703	10555	10555
query52	87	87	76	76
query53	254	268	201	201
query54	273	222	206	206
query55	73	74	66	66
query56	300	309	274	274
query57	1288	1305	1196	1196
query58	279	274	257	257
query59	1571	1676	1418	1418
query60	316	272	251	251
query61	154	149	150	149
query62	567	495	422	422
query63	236	197	195	195
query64	2825	1070	871	871
query65	4610	4478	4639	4478
query66	1827	492	366	366
query67	29429	29324	29105	29105
query68	3141	1583	981	981
query69	428	307	256	256
query70	945	816	805	805
query71	368	331	307	307
query72	3037	2690	2310	2310
query73	845	827	445	445
query74	5066	4877	4704	4704
query75	2532	2499	2148	2148
query76	2310	1158	794	794
query77	331	362	264	264
query78	11911	12016	11278	11278
query79	1364	1123	748	748
query80	1317	546	480	480
query81	541	333	287	287
query82	652	151	120	120
query83	382	327	290	290
query84	280	160	129	129
query85	965	622	547	547
query86	402	251	241	241
query87	1831	1829	1769	1769
query88	3700	2792	2746	2746
query89	427	384	322	322
query90	2007	202	194	194
query91	203	193	165	165
query92	61	59	55	55
query93	1648	1592	1012	1012
query94	704	351	313	313
query95	802	499	554	499
query96	1124	847	365	365
query97	2630	2615	2506	2506
query98	226	206	199	199
query99	1110	1114	966	966
Total cold run time: 266023 ms
Total hot run time: 177617 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

query1	0.01	0.00	0.00
query2	0.12	0.04	0.04
query3	0.25	0.13	0.13
query4	1.61	0.14	0.13
query5	0.24	0.21	0.21
query6	1.24	1.12	1.06
query7	0.03	0.00	0.00
query8	0.05	0.03	0.03
query9	0.37	0.31	0.30
query10	0.54	0.54	0.55
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.48	0.48	0.48
query14	1.01	1.03	1.01
query15	0.61	0.60	0.60
query16	0.31	0.32	0.31
query17	1.10	1.11	1.09
query18	0.22	0.21	0.20
query19	2.05	1.96	1.92
query20	0.02	0.01	0.01
query21	15.45	0.21	0.14
query22	4.82	0.06	0.05
query23	16.10	0.31	0.12
query24	2.95	0.45	0.32
query25	0.11	0.04	0.03
query26	0.74	0.21	0.16
query27	0.04	0.04	0.04
query28	3.48	0.91	0.55
query29	12.52	4.20	3.30
query30	0.27	0.16	0.17
query31	2.78	0.61	0.32
query32	3.22	0.60	0.47
query33	3.13	3.14	3.18
query34	15.59	4.24	3.51
query35	3.54	3.49	3.48
query36	0.56	0.43	0.42
query37	0.10	0.06	0.07
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.19	0.16	0.14
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.48 s
Total hot run time: 24.9 s

@Gabriel39
Gabriel39 marked this pull request as ready for review July 30, 2026 01:39
@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner July 30, 2026 01:39
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full authoritative 22-file bundle; no additional user focus was supplied.

The four inline P1 findings are distinct remaining routes around the new effective-option guard: a read-only system-table wrapper hides its option-bearing data table, row-count planning runs outside PaimonScanNode, partition snapshot loading scans a raw physical handle before overrides, and a fallback reader exposes only the main branch's options.

Checkpoint conclusions:

  • Goal and tests: direct data-table option precedence and both JNI scanner protocols are covered, but the changed tests do not exercise the four real wrapper/pre-planning/fallback routes above.
  • Scope, parallel paths, and special conditions: V1/V2 JNI framing, data/system/incremental/options/no-options paths, empty and nested identifiers, partitioned metadata, row-count loading, and fallback branches were traced; only the four inline paths remain defective.
  • Concurrency, lifecycle, persistence, and compatibility: detached catalog ALTER validation, single live publication/reset, replay/image compatibility, and legacy-key filtering are sound; no additional race or upgrade defect remains.
  • Configuration and performance: valid catalog/relation precedence works on the foreground handle, but the four missed manifest paths can still block on a zero-permit executor or mutate Paimon's JVM-global manifest pool.
  • Observability and FE/BE propagation: the credential-bearing raw parameter log is removed, the cached scanner metric is safely published, and the paired encoded schema plus serialized processed-table handoff are consistent.
  • Data writes and transactions: no data-write/transaction behavior is changed by this PR, and no additional persistence issue was found.

Validation was static only. Per the authoritative bundle, no builds, tests, source changes, or commits were performed.

Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonScanParams.java Outdated
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Paimon read-only system tables and fallback tables hide the file-store handles that actually plan manifests, while snapshot partition loading and row-count collection can plan before ScanNode. An unsafe physical manifest parallelism could therefore bypass the final visible-table check, and early validation could also reject a safe relation-level override. Validate each effective planning handle, preserve the fully merged table during partition projection, and guard row-count plans before they start.

### Release note

Paimon manifest parallelism safeguards now cover system-table, fallback-branch, partition-metadata, and statistics planning paths.

### Check List (For Author)

- Test: FE checkstyle, Groovy compilation, and manual Paimon 1.3.1 fixture probe passed; unit and P0 regression coverage added but not run locally because the installed third-party toolchain is absent
- Behavior changed: Yes. Unsafe hidden manifest-planning handles are rejected before planning while safe relation overrides remain effective.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29472 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ea521522915c65fcbcf1239de94d125d89c0ff9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17594	4220	4110	4110
q2	2035	326	197	197
q3	10281	1512	839	839
q4	4710	478	341	341
q5	7619	848	575	575
q6	211	173	139	139
q7	774	833	615	615
q8	10045	1615	1668	1615
q9	5775	4337	4340	4337
q10	6902	1765	1461	1461
q11	502	351	328	328
q12	743	577	458	458
q13	18107	3272	2678	2678
q14	267	263	241	241
q15	q16	773	784	727	727
q17	1046	1034	952	952
q18	6887	5918	5627	5627
q19	1322	1373	1125	1125
q20	804	683	542	542
q21	5639	2590	2249	2249
q22	434	359	316	316
Total cold run time: 102470 ms
Total hot run time: 29472 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4405	4303	4310	4303
q2	283	318	211	211
q3	4580	4956	4368	4368
q4	2057	2138	1371	1371
q5	4404	4244	4333	4244
q6	229	186	132	132
q7	1714	2053	1755	1755
q8	2451	2170	2112	2112
q9	7878	7750	7795	7750
q10	4693	4689	4245	4245
q11	572	421	384	384
q12	734	874	624	624
q13	3196	3510	3042	3042
q14	286	308	283	283
q15	q16	742	749	654	654
q17	1367	1332	1330	1330
q18	8035	7412	6957	6957
q19	1126	1114	1097	1097
q20	2242	2215	1939	1939
q21	5234	4603	4446	4446
q22	525	459	410	410
Total cold run time: 56753 ms
Total hot run time: 51657 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review summary

Requesting changes for four substantiated issues at 8ea521522915c65fcbcf1239de94d125d89c0ff9: three P1 correctness/compatibility gaps in legacy STRUCT parsing and effective Paimon wrapper validation, plus one P2 repeated partition-enumeration regression for reader-only OPTIONS.

Critical checkpoint conclusions:

  1. Goal and proof — The guardrail, precedence, JNI framing, and credential-safe diagnostics goals are coherent and mostly implemented. The added FE/BE/P0 tests are broad, but they do not prove the four reported cases.
  2. Scope and clarity — Most changes are focused and reuse common helpers. The shared ColumnType parser change unintentionally broadens the compatibility impact beyond Paimon, and the OPTIONS projection path does redundant work.
  3. Concurrency — Detached catalog validation, synchronized publication/reset, double-checked system-table initialization, and volatile/synchronized counter sampling were traced. No lock-order or publication defect remains beyond the independently cached handle lifecycle reported inline.
  4. Lifecycle — Scanner open/close and catalog reset lifecycles are balanced. The source-table cache and system-wrapper cache can nevertheless represent different generations, which is reported inline.
  5. Configuration — Reader option ranges and relation > catalog > physical precedence are generally enforced, including CREATE/ALTER/replay paths. Privilege-wrapped fallback and divergent system-wrapper children bypass the intended complete effective-table validation.
  6. Compatibility — The paired Base64 schema payload is emitted by both V1 and V2 Paimon readers and the encoded protocol is rolling-compatible. The unencoded public parser no longer preserves its prior lowercase STRUCT-key contract.
  7. Parallel paths — Normal, snapshot, tag, branch, startup, incremental, row-count/statistics, partition, system-table, fallback, V1, and V2 paths were traced. The privilege-wrapped fallback topology is the remaining missed path.
  8. Conditional checks — New fail-closed checks are generally documented and correctly placed; the direct instanceof FallbackReadFileStoreTable condition is insufficient for supported delegated wrappers.
  9. Test coverage — Positive and negative unit/P0 coverage exists for ranges, precedence, lifecycle, framing, fallback, system tables, and row-count planning. Missing coverage corresponds to the four inline findings.
  10. Test results — The changed expected behavior is internally consistent on inspection. No tests or builds were run because the authoritative review prompt prohibits them.
  11. Observability — Raw scanner parameters are no longer logged; the derived debug summary and counters are appropriately bounded. No additional logging/metrics issue was found.
  12. Persistence and transactions — Detached ALTER validation, EditLog publication/replay, restart behavior, and legacy property filtering were traced without a separate persistence defect. No data-write transaction path is changed.
  13. FE/BE propagation — Both BE reader generations publish the new paired fields and the Java scanner consumes them consistently. The compatibility problem is confined to the shared legacy parser behavior described inline.
  14. Performance — Reader-only OPTIONS unnecessarily bypass the memoized latest projection and re-enumerate partitions; no other material CPU, allocation, or memory-accounting regression was substantiated.
  15. Other issues — External catalog input is trusted under the repository threat model; no separate security vulnerability was identified. All other candidates were either fixed in existing live threads or dismissed with concrete code evidence.

There was no additional user-provided review focus, so the whole PR was reviewed. All required second-round normal and risk reviewers converged on this exact four-comment set with NO_NEW_VALUABLE_FINDINGS.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177744 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ea521522915c65fcbcf1239de94d125d89c0ff9, data reload: false

query5	4327	635	478	478
query6	462	221	202	202
query7	4912	626	336	336
query8	344	189	179	179
query9	8774	4105	4092	4092
query10	482	358	319	319
query11	5945	2323	2133	2133
query12	158	103	106	103
query13	1264	617	450	450
query14	6247	5222	4873	4873
query14_1	4239	4229	4198	4198
query15	217	209	203	203
query16	1022	470	468	468
query17	1143	711	585	585
query18	2438	468	359	359
query19	209	189	150	150
query20	113	111	109	109
query21	247	161	137	137
query22	13543	13572	13411	13411
query23	17367	16480	16166	16166
query23_1	16241	16341	16290	16290
query24	7491	1763	1302	1302
query24_1	1310	1304	1270	1270
query25	577	461	398	398
query26	1328	361	222	222
query27	2593	607	382	382
query28	4432	2042	2074	2042
query29	1095	640	493	493
query30	347	275	230	230
query31	1122	1111	988	988
query32	111	64	62	62
query33	550	329	265	265
query34	1181	1098	665	665
query35	765	779	667	667
query36	1033	1019	912	912
query37	162	118	103	103
query38	1886	1700	1659	1659
query39	881	883	856	856
query39_1	842	848	870	848
query40	243	161	144	144
query41	66	63	65	63
query42	89	93	89	89
query43	321	320	281	281
query44	1398	772	773	772
query45	196	183	170	170
query46	1042	1200	734	734
query47	2122	2132	2050	2050
query48	398	416	303	303
query49	583	415	309	309
query50	1097	427	345	345
query51	10404	10793	10699	10699
query52	84	86	76	76
query53	273	279	196	196
query54	275	227	222	222
query55	73	72	65	65
query56	280	301	290	290
query57	1307	1289	1171	1171
query58	286	266	269	266
query59	1607	1657	1427	1427
query60	329	267	256	256
query61	149	154	153	153
query62	543	500	433	433
query63	248	198	202	198
query64	2856	1077	915	915
query65	4752	4665	4656	4656
query66	1837	495	380	380
query67	29484	29279	29139	29139
query68	3074	1537	1023	1023
query69	407	318	265	265
query70	889	798	808	798
query71	381	355	321	321
query72	3086	2680	2320	2320
query73	827	764	432	432
query74	5068	4910	4796	4796
query75	2525	2493	2148	2148
query76	2334	1195	778	778
query77	361	370	282	282
query78	12005	11921	11265	11265
query79	1421	1241	762	762
query80	1311	544	460	460
query81	521	331	284	284
query82	634	154	118	118
query83	360	327	293	293
query84	280	164	132	132
query85	964	625	516	516
query86	404	244	233	233
query87	1821	1818	1746	1746
query88	3734	2822	2815	2815
query89	448	400	326	326
query90	1901	196	193	193
query91	203	190	161	161
query92	62	60	59	59
query93	1622	1504	908	908
query94	715	362	288	288
query95	776	515	566	515
query96	1153	875	360	360
query97	2660	2602	2505	2505
query98	213	216	198	198
query99	1104	1128	975	975
Total cold run time: 263379 ms
Total hot run time: 177744 ms

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: The unencoded JNI STRUCT parser no longer preserved its historical lowercase field-key contract. Paimon fallback validation also stopped at privilege delegates, system tables could be built from a separately reloaded data handle, and reader-only relation options unnecessarily bypassed the memoized partition projection. Restore legacy parser compatibility, traverse delegated planning handles, construct system wrappers from the exact validated source handle, and keep pure reader tuning on the cached projection.

### Release note

Paimon reader tuning avoids redundant partition enumeration, privilege-wrapped fallback tables receive complete manifest validation, and legacy unencoded JNI STRUCT keys retain lowercase compatibility.

### Check List (For Author)

- Test: PaimonJniScannerTest (15 passed), targeted legacy parser test, Paimon 1.3.1 privilege-wrapper probe, FE checkstyle, and diff checks passed; FE unit coverage added but not run locally because the installed third-party toolchain is absent
- Behavior changed: Yes. System wrappers share the validated source handle and pure reader options reuse memoized metadata.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29255 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43f91409ddca4dbf0d298b49ea14cec0da617b5d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17629	4108	4109	4108
q2	2018	326	198	198
q3	10274	1414	842	842
q4	4676	479	340	340
q5	7474	839	577	577
q6	182	167	133	133
q7	736	825	619	619
q8	9360	1451	1530	1451
q9	5644	4311	4291	4291
q10	6842	1725	1497	1497
q11	495	375	313	313
q12	711	590	452	452
q13	18099	3500	2796	2796
q14	271	262	242	242
q15	q16	789	777	704	704
q17	979	865	1036	865
q18	6774	5862	5554	5554
q19	1294	1248	944	944
q20	757	678	571	571
q21	5959	2555	2460	2460
q22	423	363	298	298
Total cold run time: 101386 ms
Total hot run time: 29255 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4344	4288	4295	4288
q2	290	306	206	206
q3	4602	4917	4389	4389
q4	2033	2144	1335	1335
q5	4377	4278	4245	4245
q6	236	176	126	126
q7	1709	1606	1811	1606
q8	2636	2235	2145	2145
q9	7965	8103	7693	7693
q10	4677	4638	4236	4236
q11	556	428	379	379
q12	748	759	542	542
q13	3178	3625	2907	2907
q14	315	305	281	281
q15	q16	730	726	639	639
q17	1352	1304	1300	1300
q18	8107	7388	7155	7155
q19	1158	1128	1123	1123
q20	2211	2196	1923	1923
q21	5190	4503	4397	4397
q22	502	434	415	415
Total cold run time: 56916 ms
Total hot run time: 51330 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177329 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43f91409ddca4dbf0d298b49ea14cec0da617b5d, data reload: false

query5	4319	626	496	496
query6	471	226	201	201
query7	4839	614	341	341
query8	341	191	169	169
query9	8772	4151	4158	4151
query10	473	362	293	293
query11	5940	2326	2157	2157
query12	164	101	99	99
query13	1289	608	441	441
query14	6234	5165	4894	4894
query14_1	4218	4182	4221	4182
query15	212	203	176	176
query16	1022	499	435	435
query17	897	695	556	556
query18	2433	473	337	337
query19	232	188	151	151
query20	108	104	108	104
query21	235	156	132	132
query22	13550	13479	13341	13341
query23	17384	16483	16080	16080
query23_1	16237	16246	16151	16151
query24	7454	1724	1285	1285
query24_1	1334	1286	1280	1280
query25	537	444	368	368
query26	1331	384	225	225
query27	2579	590	389	389
query28	4526	2045	2052	2045
query29	1092	638	498	498
query30	354	268	228	228
query31	1125	1095	1001	1001
query32	109	61	64	61
query33	548	319	258	258
query34	1175	1136	666	666
query35	765	776	670	670
query36	1019	1048	897	897
query37	151	109	97	97
query38	1878	1720	1680	1680
query39	885	904	858	858
query39_1	823	860	835	835
query40	260	170	143	143
query41	71	69	72	69
query42	96	96	97	96
query43	330	334	285	285
query44	1416	770	774	770
query45	197	177	176	176
query46	1091	1180	743	743
query47	2151	2156	2012	2012
query48	416	410	297	297
query49	590	451	319	319
query50	1089	441	335	335
query51	10609	10707	10524	10524
query52	88	84	77	77
query53	274	279	208	208
query54	292	263	233	233
query55	78	89	67	67
query56	316	302	308	302
query57	1326	1295	1198	1198
query58	307	291	266	266
query59	1595	1667	1433	1433
query60	329	284	272	272
query61	179	172	172	172
query62	543	497	441	441
query63	254	209	202	202
query64	2889	999	863	863
query65	4764	4593	4617	4593
query66	1841	511	392	392
query67	29291	29232	29083	29083
query68	3134	1585	1046	1046
query69	413	310	291	291
query70	922	826	819	819
query71	363	327	319	319
query72	3005	2677	2401	2401
query73	786	768	435	435
query74	5058	4943	4747	4747
query75	2522	2514	2111	2111
query76	2332	1141	776	776
query77	340	381	279	279
query78	11685	11818	11343	11343
query79	1361	1218	757	757
query80	666	572	484	484
query81	470	329	294	294
query82	564	154	121	121
query83	407	317	296	296
query84	327	156	133	133
query85	951	618	516	516
query86	323	283	230	230
query87	1814	1848	1761	1761
query88	3704	2868	2811	2811
query89	446	385	341	341
query90	1914	194	196	194
query91	203	194	170	170
query92	63	63	53	53
query93	1630	1506	902	902
query94	548	342	315	315
query95	792	597	479	479
query96	1065	848	349	349
query97	2639	2596	2466	2466
query98	217	204	202	202
query99	1086	1105	978	978
Total cold run time: 261824 ms
Total hot run time: 177329 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43f91409ddca4dbf0d298b49ea14cec0da617b5d, data reload: false

query1	0.01	0.00	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.13	0.13
query5	0.25	0.23	0.24
query6	1.19	0.85	0.85
query7	0.04	0.00	0.00
query8	0.06	0.04	0.03
query9	0.40	0.32	0.32
query10	0.57	0.59	0.62
query11	0.20	0.14	0.14
query12	0.18	0.14	0.14
query13	0.49	0.47	0.48
query14	1.04	1.01	1.00
query15	0.64	0.59	0.60
query16	0.32	0.33	0.33
query17	1.10	1.16	1.16
query18	0.23	0.23	0.22
query19	2.06	1.99	1.92
query20	0.02	0.02	0.02
query21	15.44	0.22	0.13
query22	4.78	0.05	0.06
query23	16.15	0.31	0.13
query24	2.97	0.42	0.32
query25	0.10	0.05	0.04
query26	0.73	0.20	0.14
query27	0.04	0.04	0.03
query28	3.57	0.89	0.56
query29	12.48	4.13	3.30
query30	0.27	0.16	0.15
query31	2.77	0.60	0.32
query32	3.23	0.62	0.50
query33	3.17	3.24	3.15
query34	15.59	4.26	3.54
query35	3.52	3.52	3.53
query36	0.56	0.46	0.43
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.20	0.16	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.66 s
Total hot run time: 24.99 s

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two correctness regressions still violate the advertised relation > catalog > physical option precedence, and two P2 issues remain in shared-reader performance and end-to-end test proof.

Critical checkpoint conclusions:

  • Goal and proof: The JNI framing, option bounds, detached catalog validation, and observability work is mostly in place, but the goal is not complete while safe overrides fail at the descriptor and memoized-projection boundaries. The new P0 success cases also do not prove returned values.
  • Scope and focus: The 31-file change is broadly cohesive across native producers, Java consumers, FE planning, catalog/cache lifecycle, and tests. The shared V2 placement called out inline is the one unfocused cross-connector cost.
  • Concurrency and lifecycle: Catalog ALTER publication is serialized, detached validation does not publish rejected state, initialization/reset is monitor-protected, scanner cleanup balances active metrics, and the cached thread counter has safe synchronized/volatile publication. I found no new deadlock, leak, or static-initialization issue.
  • Configuration and compatibility: Accepted catalog changes reset live derived state and the Paimon cache. Both V1 and V2 retain the legacy schema fields while publishing the encoded pair, and the new Java consumer rejects mixed framing; rolling compatibility and recursive STRUCT handling otherwise check out. Dynamic relation precedence is incomplete because of the two P1 findings.
  • Parallel paths and conditions: I traced data/system tables, fallback/delegated/privilege wrappers, memoized/direct projections, row counts/statistics, ScanNode planning, descriptor serialization, and all six V2 JNI subclasses. The two P1 comments are distinct later/composed instances, not duplicates of the existing threads.
  • Tests and results: Unit and negative coverage is broad, but deterministic successful P0 cases use success-only test blocks and add no generated .out. This automated pass ran no builds or tests because the authoritative review contract forbids them; the checkout also lacks .worktree_initialized and thirdparty/installed.
  • Observability and persistence: The changed constructor log is credential-safe, JVM-global gauges use set semantics, and CREATE/ALTER/reject/replay/image reconstruction preserve the expected journaled/raw property state. No additional logging, metric, replay, or failover issue was found.
  • Data/protocol/performance: No data-write, transaction-visibility, delete-bitmap, or storage-format path changes here. Both native scanner paths transmit the new internal pair. The remaining performance issue is the Paimon-only per-split work in the shared V2 base.

User focus: review_focus.txt contains no additional user-provided focus, so the full PR was reviewed. Three review rounds converged on this exact four-comment set, with every final normal and risk-focused reviewer returning NO_NEW_VALUABLE_FINDINGS.

Comment thread be/src/format_v2/jni/jni_table_reader.cpp Outdated
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Relation-safe Paimon overrides could be rejected later when system-table descriptor serialization rebuilt schema from the unsafe physical handle or when memoized partition metadata revalidated reader-only physical settings. The shared V2 JNI path also encoded Paimon-only schema payloads for unrelated connectors, and the P0 success paths did not assert returned values. Keep schema serialization non-planning, restrict metadata validation to manifest planning settings, publish encoded schema only for Paimon, and add result-based coverage including the partitioned safe-override composition.

### Release note

Paimon relation overrides remain effective through metadata and descriptor phases, while unrelated V2 JNI connectors avoid Paimon schema-encoding work.

### Check List (For Author)

- Test: PaimonJniScannerTest (15 passed), FE checkstyle, regression framework compile, C++ format check, and diff checks passed. FE/BE unit and external Paimon regression coverage were added but not run locally because the required third-party toolchain is absent.
- Behavior changed: Yes. Metadata-only paths no longer reject reader-only physical values after a safe relation override, and encoded schema publication is Paimon-specific.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29823 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f07b707b698b7229d4f2826298e6790e5d19802, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17618	4302	4195	4195
q2	2157	325	203	203
q3	10126	1465	823	823
q4	4685	475	338	338
q5	7495	866	581	581
q6	185	171	137	137
q7	793	815	609	609
q8	9356	1543	1482	1482
q9	5654	4348	4357	4348
q10	6790	1720	1491	1491
q11	512	357	318	318
q12	756	587	467	467
q13	18165	3528	2785	2785
q14	262	266	251	251
q15	q16	795	774	712	712
q17	1054	1032	1001	1001
q18	6951	5832	5598	5598
q19	1308	1368	1051	1051
q20	819	681	566	566
q21	6135	2727	2565	2565
q22	449	356	302	302
Total cold run time: 102065 ms
Total hot run time: 29823 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4445	4409	4320	4320
q2	295	346	210	210
q3	4588	4982	4404	4404
q4	2086	2191	1372	1372
q5	4441	4311	4299	4299
q6	241	177	132	132
q7	2036	2091	1648	1648
q8	2586	2213	2239	2213
q9	8110	8015	7774	7774
q10	4687	4713	4239	4239
q11	578	435	408	408
q12	871	781	540	540
q13	3207	3629	2942	2942
q14	301	301	270	270
q15	q16	715	756	670	670
q17	1374	1346	1332	1332
q18	8022	7339	7558	7339
q19	1177	1145	1123	1123
q20	2205	2220	1957	1957
q21	5290	4639	4472	4472
q22	513	465	397	397
Total cold run time: 57768 ms
Total hot run time: 52061 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 179669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f07b707b698b7229d4f2826298e6790e5d19802, data reload: false

query5	4304	627	474	474
query6	465	230	206	206
query7	4854	585	343	343
query8	346	202	167	167
query9	8720	4179	4154	4154
query10	453	359	311	311
query11	5921	2342	2149	2149
query12	169	103	99	99
query13	1283	622	444	444
query14	6265	5246	4893	4893
query14_1	4260	4272	4240	4240
query15	204	208	179	179
query16	995	488	452	452
query17	950	757	578	578
query18	2484	486	348	348
query19	221	199	155	155
query20	114	108	110	108
query21	239	162	140	140
query22	13628	13596	13341	13341
query23	17540	16691	16241	16241
query23_1	16248	16339	16185	16185
query24	7523	1771	1313	1313
query24_1	1303	1320	1301	1301
query25	575	471	404	404
query26	1324	363	213	213
query27	2590	610	382	382
query28	4473	2062	2046	2046
query29	1091	625	486	486
query30	349	261	245	245
query31	1121	1092	976	976
query32	108	63	63	63
query33	542	325	260	260
query34	1187	1132	645	645
query35	781	805	680	680
query36	1042	1045	879	879
query37	168	109	93	93
query38	1879	1709	1656	1656
query39	907	913	841	841
query39_1	851	830	842	830
query40	262	170	145	145
query41	68	63	64	63
query42	95	94	98	94
query43	330	330	278	278
query44	1435	774	763	763
query45	192	189	171	171
query46	1066	1223	750	750
query47	2134	2088	1994	1994
query48	409	412	302	302
query49	576	411	306	306
query50	1118	429	346	346
query51	10770	10563	10646	10563
query52	89	88	74	74
query53	269	282	203	203
query54	277	235	226	226
query55	75	73	67	67
query56	306	303	292	292
query57	1334	1288	1241	1241
query58	286	265	257	257
query59	1628	1681	1462	1462
query60	332	282	264	264
query61	154	151	156	151
query62	563	513	443	443
query63	249	204	219	204
query64	2850	1047	900	900
query65	4731	4657	4667	4657
query66	1892	500	430	430
query67	29389	29333	29271	29271
query68	3152	1578	1050	1050
query69	433	309	265	265
query70	922	830	811	811
query71	377	326	340	326
query72	3099	2740	2367	2367
query73	854	826	446	446
query74	5126	4914	4743	4743
query75	2542	2504	2163	2163
query76	2340	1192	806	806
query77	350	405	286	286
query78	11988	11968	11530	11530
query79	1456	1192	769	769
query80	1308	548	474	474
query81	523	340	292	292
query82	706	156	121	121
query83	387	323	294	294
query84	323	158	135	135
query85	1012	635	510	510
query86	438	251	226	226
query87	1831	1824	1762	1762
query88	3762	2835	2808	2808
query89	461	370	334	334
query90	1977	202	194	194
query91	207	195	170	170
query92	60	59	54	54
query93	1707	1586	989	989
query94	849	356	322	322
query95	815	579	503	503
query96	1056	775	349	349
query97	2615	2631	2493	2493
query98	215	229	221	221
query99	1096	1120	976	976
Total cold run time: 264737 ms
Total hot run time: 179669 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.86 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f07b707b698b7229d4f2826298e6790e5d19802, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.06	0.05
query3	0.25	0.13	0.13
query4	1.61	0.14	0.14
query5	0.24	0.22	0.23
query6	1.17	0.82	0.80
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.38	0.30	0.30
query10	0.55	0.55	0.57
query11	0.20	0.13	0.14
query12	0.18	0.14	0.15
query13	0.47	0.47	0.47
query14	1.04	1.02	1.01
query15	0.65	0.62	0.59
query16	0.33	0.31	0.31
query17	1.10	1.08	1.12
query18	0.22	0.22	0.21
query19	2.00	1.95	1.92
query20	0.02	0.01	0.02
query21	15.43	0.21	0.13
query22	4.75	0.05	0.06
query23	16.14	0.31	0.12
query24	3.00	0.42	0.31
query25	0.12	0.05	0.05
query26	0.74	0.20	0.15
query27	0.04	0.04	0.03
query28	3.48	0.95	0.53
query29	12.48	4.13	3.32
query30	0.27	0.16	0.15
query31	2.77	0.61	0.31
query32	3.22	0.59	0.48
query33	3.25	3.28	3.27
query34	15.74	4.25	3.55
query35	3.51	3.51	3.55
query36	0.55	0.44	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.63 s
Total hot run time: 24.86 s

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for two remaining Paimon relation-option correctness gaps. The inline comments cover (1) a relation-option-bearing statement snapshot leaking into another alias of the same table and (2) pre-lock external metadata preload discarding a safe relation override before validation. No additional user-provided review focus was supplied; the complete PR was reviewed.

Critical checkpoint conclusions:

  • Goal and proof: The JNI transport, option guardrails, detached ALTER validation, log sanitization, and statistics-cache goals are mostly implemented and broadly tested, but the two planner paths above mean relation precedence/isolation is not yet complete. The missing two-alias manifest case and mixed internal/Paimon preload case are requested inline.
  • Scope and clarity: The patch is broad because it crosses C++, Java JNI, FE metadata/planning, compatibility, and tests, but the changes are focused on those required boundaries. The two incomplete parallel paths are the remaining blockers.
  • Concurrency: Detached Paimon ALTER no longer exposes rejected candidates, and the shared thread counter's volatile value-before-timestamp publication plus synchronized expiry recheck is sound. The late cache-initialization race considered during review is pre-existing and not introduced or worsened here.
  • Lifecycle: JNI scanner setup/close, cached table handles, fallback/delegated/system wrappers, row-count planning, and descriptor serialization were traced. The statement snapshot and pre-bind preload lifecycles produce the two inline issues; no additional lifecycle defect was found.
  • Configuration and dynamic changes: Catalog/relation reader options are range-validated after copies, and successful ALTER resets derived properties and invalidates relevant caches. Precedence is still lost in the two reported statement paths.
  • Compatibility: Both V1 and V2 Paimon producers publish the paired encoded schema while retaining the legacy fields. Old/new producer-consumer combinations, nested and empty identifiers, legacy lowercase STRUCT keys, malformed-pair rejection, and raw-but-inactive persisted legacy options remain compatible.
  • Parallel paths and special checks: Foreground scans, reader-only memoization, manifest/startup/branch selection, fallback and privilege delegates, system tables, row counts, descriptor serialization, replay, and both scanner routes were checked. The validation conditions are documented and explicit; pre-lock preload invokes one at the wrong lifecycle boundary.
  • Tests and results: The changed unit/P0 tests use deterministic named result assertions and negative cases, and their expected outputs were inspected. The author reports targeted scanner tests, checkstyle, format, Groovy compilation, and probes; this review ran no builds or tests because the authoritative review bundle forbids them. Coverage is missing exactly the two cases requested inline.
  • Observability: Raw credential-bearing scanner parameters are no longer logged, and the one-second JVM-global thread gauge has bounded, retryable sampling. No additional metric or logging blocker was found.
  • Persistence and transactions: Detached live validation, final publication, edit-log ordering, replay, cache reset, and legacy-option filtering remain coherent. No data-write transaction behavior is changed.
  • FE/BE propagation: The paired schema parameters are emitted by both Paimon C++ reader routes and consumed atomically by the Java scanner; no other FE/BE variable propagation gap was found.
  • Performance: Reader-only options retain the memoized projection, encoded framing is Paimon-capability-gated, and cached thread sampling removes repeated JVM thread walks. No additional actionable performance regression was found.
  • Other issues: All initial risks and one pre-existing ALTER/cache candidate were explicitly resolved. Two review rounds converged on this exact comment set at head 0f07b707b698b7229d4f2826298e6790e5d19802.

}
// The shared latest cache was built from the catalog-scoped handle. Relation options
// need their own projection so partition enumeration uses the final safe table copy.
return PaimonUtils.loadSnapshotProjection(this, effectiveTable);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep this option-bearing projection relation-scoped. StatementContext.loadSnapshots has only one MvccTableInfo slot per catalog/db/table, so if the first alias uses @options('scan.manifest.parallelism'='1'), this snapshot retains 1 and a later unannotated alias reuses it through PaimonSource; reversing alias order instead rejects the raw physical 0. That makes relation isolation and validation traversal-order-dependent. Please keep the shared latest snapshot neutral or key option projections by relation/scan parameters, and add a two-alias order test.

List<Partition> paimonPartitions = tableLoader.catalog(nameMapping).getPaimonPartitions(nameMapping);
// Catalog.listPartitions reloads the raw physical table and loses Doris catalog/relation
// copies. Enumerate the already merged handle so validation and planning see one table.
PaimonReaderOptions.validateEffectivePlanningTable(paimonTable);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the relation override during pre-lock preload. Selector-free Paimon @options are registered as a latest relation, but when preload is enabled for a mixed internal/external query, PreloadExternalMetadata calls loadSnapshots(table, empty, empty) before BindRelation. A partitioned physical table with scan.manifest.parallelism=0 therefore reaches this validator and fails before its safe relation override of 1 is applied. Please retain the scan params/effective handle in preload, or skip latest-snapshot preload for these relations, and cover the mixed-table preload path.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 71.51% (128/179) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants